Skip to content

feat(precompute): build stored summary panes from Remote Write samples - #820

Draft
zzylol wants to merge 2 commits into
feat/sdsfrom
feat/precompute-engine
Draft

zzylol wants to merge 2 commits into
feat/sdsfrom
feat/precompute-engine

Conversation

@zzylol

@zzylol zzylol commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Stack 3/5. Base: #819. Next: #821.

Before this PR

Remote Write samples were accepted and dropped.

After this PR

A precompute engine on the engine thread builds stored panes from Remote Write:

  • Selection: samples are buffered per definition and pane. The plan's own Scan selects them, so its label matchers apply.
  • Building: a pane (end − width, end] is built once the watermark (newest sample − allowed lateness) passes end. The build runs the definition's builder compiled from the DAG and stores the summary batch. Panes are open at the start, like PromQL ranges and the executor's windows.
  • Lateness and startup: a sample arriving after its pane was built is dropped and counted. The first pane after startup is skipped because it may be partial.
  • Eviction: panes older than the largest offset plus two widths are evicted.

/api/v1/write returns 503 when the engine queue is full.

Testing

data_plane/tests/precompute.rs (5 tests): the watermark, lateness, label matching, left-open boundaries, eviction.

🤖 Generated with Claude Code

zzylol and others added 2 commits October 7, 2026 03:11
An engine thread owns the installed plan (Planner IR holds Rcs) and receives
Remote Write samples from the HTTP server. For each stored summary
definition it buffers its Scan metric's samples by tumbling pane. When the
watermark (newest sample minus --allowed-lateness-ms) passes a pane's end,
it runs the plan's Scan over them (applying label matchers) and the
definition's build from its TimeRange input, then stores the output. Panes
without samples are stored empty; the first pane seen is skipped because it
may be partial; samples for built panes are dropped. A full engine queue
answers Remote Write with 503 so senders retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A PromQL range at T covers (T - range, T], and the executor's series
windows follow it. Panes were [start, end), so a sample at a pane boundary
was summarized in the next pane. Panes are now (end - width, end].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant